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1.  Introduction 


When  an  image  is  captured  by  a  monochrome  camera,  a  single  charge-coupled  device  (CCD)  or 
complementary  metal-oxide  semiconductor  (CMOS)  sensor  is  used  to  sample  the  light  intensity 
projected  onto  the  sensor.  Color  images  are  captured  in  much  the  same  way,  except  that  the  light 
intensity  is  measured  in  separate  color  bands,  usually  red,  green,  and  blue.  In  order  to  do  this, 
three  separate  sensors  could  be  used  in  conjunction  with  a  beam  splitter  to  accurately  measure 
each  of  the  three  primary  colors  at  each  pixel.  However,  this  approach  is  expensive  and 
mechanically  difficult  to  implement,  making  its  use  in  commercial  imaging  systems  infeasible. 
To  overcome  this  obstacle,  the  color  filter  array  (CFA)  was  introduced  to  capture  a  color  image 
using  only  one  sensor. 

A  CFA  is  an  array  of  alternating  color  filters  that  samples  only  one  color  band  at  each  pixel 
location.  The  most  popular  CFA  pattern  is  the  Bayer  pattern  (figure  1),  which  features  blue  and 
red  filters  at  alternating  pixel  locations  in  the  horizontal  and  vertical  directions,  and  green  filters 
organized  in  the  quincunx  pattern  at  the  remaining  locations  (1).  This  pattern  results  in  half  of 
the  image  resolution  being  dedicated  to  accurate  measurement  of  the  green  color  band.  The  peak 
sensitivity  of  the  human  visual  system  lies  in  the  medium  wavelengths,  justifying  the  extra  green 
sampling  (2).  Because  each  pixel  now  has  only  one  color  sampled,  a  demosaicing  algorithm 
must  be  employed  to  recover  the  missing  information. 


Figure  1 .  The  Bayer  pattern  features  blue  and  red  filters  at 

alternating  pixel  locations  in  the  horizontal  and  vertical 
directions  and  green  filters  organized  in  the  quincunx 
pattern  at  the  remaining  locations. 

CFA  demosaicing  is  the  process  through  which  three  fully  populated  color  planes  are  created 
from  the  CFA  data.  Several  algorithms  exist  for  this  purpose,  ranging  from  simple  linear 
interpolators  to  high-end  nonlinear  interpolators  that  exploit  as  much  spatial  and  spectral 
information  as  possible. 
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The  rest  of  the  report  is  organized  as  follows.  Several  demosaicing  algorithms  are  introdueed  in 
seetion  2,  starting  with  simple  algorithms  sueh  as  nearest  neighbor  and  ending  with  more  robust 
gradient-based  algorithms.  In  section  3,  the  false  coloring  and  zipper  effect  demosaicing  artifacts 
are  introduced  and  discussed,  after  which  two  post-processing  techniques  for  removing  false 
colors  are  introduced  in  section  4.  The  classic  performance  analysis  techniques  are  introduced 
and  discussed  in  section  5.1,  before  one  no-reference  technique  is  discussed  and  two  new 
techniques  are  proposed  in  section  5.2  in  response  to  the  frequent  lack  of  reference  data.  In 
section  6,  all  algorithms  introduced  in  section  2  are  analyzed  with  these  new  methods,  along  with 
a  wealth  of  images  to  justify  the  results. 


2.  CFA  Demosaicing 


2.1  Nearest  Neighbor 

The  simplest  of  all  interpolation  algorithms  is  nearest  neighbor  interpolation.  Using  a  2x2 
neighborhood  from  the  Bayer  pattern  CFA  (figure  2),  missing  pixel  values  are  interpolated  by 
simply  adopting  the  nearest  sampled  value  (5).  The  sampled  blue  and  red  values  in  this  2x2 
neighborhood  are  used  at  the  three  remaining  locations.  The  sampled  green  values  can  be  moved 
in  either  a  vertical  or  horizontal  direction  to  fill  in  the  pixels  without  green  information. 


Figure  2.  2x2  Bayer  CFA 
neighborhood. 

This  method  introduces  significant  color  errors,  especially  along  edges.  However,  since  no 
calculations  are  performed,  this  method  may  be  beneficial  in  applications  where  speed  is  critical, 
such  as  video  imaging  systems. 

2.2  Linear  Interpolation 

Another  simple  interpolation  algorithm  is  linear  interpolation.  A  slightly  larger  3x3 
neighborhood  is  taken  from  the  CFA  (figure  3),  and  missing  pixel  values  are  estimated  by  the 
averaging  of  nearby  values  (3).  Equations  used  in  this  interpolation  method  are  as  follows  (4): 

G5  =  ^*(G2  +  G4  +  G6  +  G8)  (1) 
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(2) 


(3) 

(4) 

by  replacing  the  B’s  with  R’s. 


55  =  ^  *  (51  +  53  +  57  +  59) 
52  =  1*  (51 +  53) 

54  =  1*(51  +  57) 

The  missing  red  values  can  be  estimated  using  the  same  equations 


Figure  3.  3x3  Bayer  CFA  neighborhood. 

One  benefit  of  this  method  is  that  it  can  be  performed  by  a  convolution  with  the  appropriate 
kernel.  Two  kernels  are  required:  one  for  estimating  the  missing  green  values  and  one  for 
estimating  missing  red/blue  values.  The  kernels  are  given  below  and  are  identified  as  Fg  for  the 
green  kernel  and  Fc  for  the  red/blue  kernel  (5,  6). 
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This  interpolation  method  performs  well  in  smooth  areas  where  the  color  changes  slowly  from 
one  to  the  next.  However,  when  performed  along  edges  where  color  changes  occur  abruptly, 
false  color  and  zipper  artifacts  are  introduced,  resulting  in  a  poor  quality  image  (see  section  3  for 
information  about  these  types  of  artifact). 

2.3  Cubic  Interpolation 

Similar  in  nature  to  the  previously  described  linear  interpolation  is  cubic  interpolation.  The 
algorithm  is  performed  with  a  simple  convolution  of  a  color  channel  with  the  appropriate  kernel. 
These  kernels  are  identified  as  follows,  where  Fg  represents  the  green  channel  kernel  and  Fc 
represents  the  red/blue  channel  kernel  (6). 
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"0  0  0  1  0  0  0“ 

0  0  -9  0  -9  0  0 

,  0  -9  0  81  0  -9  0 

Fg  = - *  1  0  81  256  81  0  1  (6) 

256  0  -9  0  81  0  -9  1 

0  0  -9  0  -9  0  0 

0  0  0  1  0  0  0_ 

"1  0  -9  -16  -9  0  1  " 

0  0  0  0  0  0  0 

_9  0  81  144  81  0  -9 

Fc  = - *  -16  0  144  256  144  0  -16  (7) 

256  _9  0  81  144  81  0  -9 

0  0  0  0  0  0  0 

1  0  -9  -16  -9  0  1 

Cubic  interpolation  suffers  from  the  same  artifacts  as  linear  interpolation,  albeit  to  a  lesser 
degree.  The  expanded  7x7  neighborhood  reduces  the  appearanee  of  these  artifacts,  but  they  are 
still  present  in  the  final  image. 

2.4  High  Quality  Linear  Interpolation 

Another  form  of  linear  interpolation,  proposed  by  Malvar  et  al.  (7),  expands  and  improves  linear 
interpolation  by  exploiting  interchannel  eorrelations  between  the  different  color  channels.  A  5x5 
neighborhood  is  used,  wherein  the  nearby  pixels  of  the  corresponding  eolor  eharmel  are  averaged 
and  then  added  to  a  correetion  term  caleulated  from  information  in  a  different  color  channel. 
Despite  only  a  modest  increase  in  the  number  of  eomputations  performed  eompared  to  the  linear 
and  eubie  interpolations,  this  method  was  shown  to  outperform  many  more  eomplicated, 
nonlinear  methods,  with  greatly  redueed  edge  artifacts.  However,  we  deeided  not  test  this 
method  in  our  study. 

2.5  Smooth  Hue  Transition  Interpolation 

Hue  is  defined  “as  the  property  of  colors  by  which  they  ean  be  perceived  as  ranging  from  red 
through  yellow,  green,  and  blue,  as  determined  by  the  dominant  wavelength  of  light  (4).”  The 
key  assumption  is  that  hue  is  smoothly  changing  across  an  object’s  surface.  The  false  eolor 
artifaet  of  linear  and  other  methods  of  interpolation  result  when  this  hue  changes  abruptly,  sueh 
as  near  an  edge.  In  this  case,  hue  is  defined  as  the  ratio  between  color  channels,  in  partieular  the 
ratio  between  red/blue  and  green  (3).  Referring  to  figure  3,  the  equations  for  interpolating  the 
blue  ehannel  are  as  follows: 


B2 


G2jB\  53^ 


(8) 


B4 


^4—  — 1 


(9) 
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(10) 


G5jB\  B3  B1  B9^ 

—  *  —  +  —  +  —  +  — 

4  [gI  G3  G1  G9) 


Equations  for  red  interpolation  are  defined  analogously.  Note  that  this  method  requires  a  fully 
populated  green  channel,  as  the  green  values  at  every  pixel  are  used.  Thus,  the  green  channel 
must  first  be  fully  interpolated  by  linear,  cubic,  or  some  other  interpolation  method  (5,  4). 

One  issue  with  this  method  is  that  it  fails  when  a  pixel  has  a  green  value  of  zero  (4).  To  resolve 
this,  a  normalized  approach  was  proposed  by  Lukac  et  al.  (5).  The  transformation  from  the 
earlier  color  ratio  is  defined  as 


B  B  +  B 

—  - — 

G  G  +  p 


(11) 


where  p  is  a  non-negative  number.  This  generalizes  to  the  earlier  definition  for  the  case  p=0.  In 
addition  to  preventing  division  errors,  experimental  evidence  shows  that  this  normalized  model 
can  further  improve  the  demosaicing  result  in  terms  of  the  mean  square  error  (MSB)  and  peak 
signal-to-noise  ratio  (PSNR)  by  increasing  the  value  of  p  (5).  For  most  cases,  p=128  produces 
sufficient  results.  In  our  tests,  we  use  p=128. 

2.6  Pattern  Recognition  Interpolation 

Thus  far,  all  of  the  interpolation  algorithms  discussed  have  had  flaws  estimating  colors  on  or 
around  edges.  In  attempt  to  counteract  this  defect,  Cok  (P)  describes  a  way  to  classify  and 
interpolate  three  different  edge  types  in  the  green  color  plane.  Once  the  green  plane  is 
interpolated,  the  red  and  blue  color  planes  are  interpolated  using  the  smooth  hue  transition 
interpolation  described  previously. 

The  first  step  in  his  procedure  is  to  find  the  average  of  the  four  neighboring  green  pixels,  and 
classify  the  neighbors  as  either  high  or  low  in  comparison  to  this  average.  For  simplicity, 
equality  is  labeled  as  a  high  value.  This  pixel  is  then  defined  as  an  edge  if  three  neighbor  pixels 
share  the  same  classification.  If  not,  then  the  pixel  can  either  be  a  part  of  a  comer  or  a  stripe.  If 
two  adjacent  neighbor  pixels  have  the  same  classification,  then  the  pixel  is  a  comer.  If  two 
opposite  pixels  have  the  same  classification,  then  the  pixel  is  a  stripe.  See  figure  4  for  a  picture 
of  each. 


Figure  4.  Example  neighborhoods  for  pattern  recognition  interpolation,  (a)  is  a  high  edge 
pattern,  (b)  is  a  low  edge  pattern,  (c)  is  a  comer  pattern,  and  (d)  is  a  stripe  pattern. 
Remember  that  H  represents  pixels  greater  than  or  equal  to  the  average  in  the  3x3 
neighborhood  and  L  represents  pixels  less  than  the  average. 
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In  the  case  of  both  types  of  edge  pattern,  the  median  of  the  neighbors  is  calculated  and  used  as 
the  missing  green  value.  For  both  the  stripe  and  comer  patterns,  more  information  must  be 
collected  from  the  surrounding  pixels.  Furthermore,  Cok  defined  the  following  clip  function  for 
use  in  the  calculation  of  the  missing  green  value  at  stripe  and  comer  locations  (P): 


clip^  (x)  = 


B 

X 

C 


X  >  B 
C<x<B 
X  <  C 


(12) 


where  B  and  C  are  the  values  corresponding  to  A>B>C>D,  the  rank  ordering  of  the  neighboring 
green  values  (3,  9).  In  both  the  stripe  and  comer  cases,  the  missing  green  value  is  defined  as 

G  =  diplilM  -  S)  (13) 


where  M  is  the  median  of  the  H  and  L  pixels,  and  S  is  the  average  of  the  X  pixels  in  the 
neighborhoods  identified  in  figure  5  (5,  9). 


As  compared  to  previous  methods,  this  interpolation  algorithm  better  preserves  edge  details  and 
minimizes  the  amount  of  zipper  artifacts  along  edges. 


Figure  5.  Neighborhoods  for  the  comer  and  stripe  patterns:  (a)  is  the 

neighborhood  for  the  comer  pattern  and  (b)  is  the  neighborhood 
for  the  stripe  pattern. 


2.7  Adaptive  Color  Plane  Interpolation 

Up  to  this  point,  the  interpolation  of  the  green  color  plane  has  occurred  using  only  information 
from  the  green  samples  from  the  CFA  data.  However,  certain  assumptions  can  be  made 
regarding  the  correlation  between  the  color  planes.  One  well-known  assumption  is  that  the  color 
planes  are  perfectly  correlated  in  a  small  enough  neighborhood.  That  is,  in  a  small  enough 
neighborhood,  the  equations 


G  =  B  +  k 
G  =  R  +  j 


(14) 
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are  true  for  eonstants  k,j.  With  this  assumption,  estimations  ean  be  made  for  the  second  derivate 
of  G  at  a  pixel  in  the  vertical  and  horizontal  directions  {4).  Local  G  information  can  also  be  used 
to  estimate  the  first  derivate  of  G  in  the  same  directions,  yielding  the  following  classifiers  for  the 
neighborhood  shown  in  figure  6  (75): 


V  =  \G2-  G8|  +  \2*B5-B\-  B9\ 
H  =  \GA-G6\  +  \2*B5-Bl>-Bl\ 


Figure  6.  Neighborhood  of  a  B  pixel  for 

adaptive  color  plane  interpolation. 


(15) 


The  same  equations  are  used  for  classifiers  at  an  R  pixel  by  simply  replacing  B  with  R.  Each  of 
these  classifiers  is  used  to  sense  high  frequency  data  in  its  corresponding  direction.  Since 
interpolating  across  an  edge  is  associated  with  artifacts  such  as  zippering,  the  goal  is  to  instead 
interpolate  along  the  edge,  that  is,  the  direction  that  is  changing  more  slowly  (70).  Thus,  the 
missing  G  value  is  determined  as  follows  {4,  10): 


G2  +  G8 

2*55-51-59 

V<H 

- + - 

2 

2 

G4  +  G6 

2*55-53-57 

V>H 

- + - 

2 

2 

G2  +  G4  +  G6  +  G8 

4*55-51-53-57-59 

V  =  H 

- + - 

4  4 


(16) 


Once  the  green  plane  is  fully  interpolated,  the  red  and  blue  planes  are  interpolated  next.  When 
interpolating  a  missing  B  value  at  an  R  pixel,  classifiers  similar  to  those  used  for  G  interpolation 
are  used.  With  figure  3  representing  the  relevant  neighborhood,  the  classifiers  are  (70) 


a  =  \B\-  B9\  + 12  *  G5  -  G1  -  G9| 
/7  =  |53-57|  +  |2*G5-G3-G7| 


(17) 
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Using  these  classifiers,  B  interpolation  is  as  follows  {10): 


B5  = 


51  +  59  2  *  G5  -  G1  -  G9 

- 1 - 

2  2 

53  +  57  2  *  G5  -  G3  -  G7 

■  + - 


a  <  p 
a>  P 


2  2 
51  +  53  +  57  +  59  4*  G5  -  G1  -  G3  -  G7  -  G9 

- + -  a  =  P 


51  +  53  2*G2-G1-G3 

52  = - + - 


(18) 


51  +  57  2*G4-G1-G7 

54  = - + - 


Interpolation  of  the  red  color  plane  is  done  in  a  similar  manner  as  for  the  blue  color  plane. 

2.8  Directionally  Weighted  Gradient  Based  Interpolation 

As  mentioned  in  section  2.7,  it  is  best  to  interpolate  missing  pixel  values  along  edges,  rather  than 
across  them.  In  order  to  expand  the  edge  detection  power  of  the  adaptive  color  plane  method,  it 
is  prudent  to  consider  more  than  two  directions.  In  some  methods,  as  many  as  12  directions  are 
considered  (11-13),  in  which  all  the  G  information  in  a  5x5  neighborhood  (shown  in  figure  7)  is 
used.  For  the  purposes  of  this  report,  four  directions  are  considered.  They  are  labeled  N,  S,  E, 
and  W,  as  indicated  in  figure  8. 


Figure  7.  5x5  Bayer  CFA  neighborhood. 


Figure  8.  5x5  Bayer  CFA  neighborhood 

with  four  cardinal  directions  labeled. 
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The  key  distinction  of  this  method  between  others  is  that  a  green  value  and  an  associated  weight 
are  assigned  to  each  of  these  four  directions.  Also,  a  gradient  is  calculated  for  each  of  these 
directions  based  on  estimations  of  the  first  order  derivatives  of  the  green  and  blue  color  planes  at 
the  pixel  under  consideration.  Using  these  gradient  values,  the  direction  weights  are  assigned  so 
as  to  be  inversely  proportional  to  the  gradient  in  the  associated  direction  (11-14). 

Using  figure  6  as  reference,  the  equations  for  calculating  the  gradients  of  G5  are  as  follows: 

=  \GS-G2\  +  \B5-Bl\ 

G.  =  \G4-G6\  +  \B5-B7\ 

=|G2-G8|  +  |55-59| 

G^  =|G6-G4|  +  |55-53| 

The  next  step  in  the  procedure  is  to  calculate  the  directional  weights: 

Dirs  =  {N,E,S,W} 

1  ,  ^  (2) 

CO.  = -  d  E  Dirs 

l  +  G, 

Since  it  is  certainly  possible  for  a  gradient  to  be  zero  (representing  an  estimated  no  change 
in  the  associated  direction),  a  one  is  added  in  the  denominator  of  coato  prevent  division 
by  zero  (11-14). 


The  previous  equations  include  a  correction  term  calculated  from  values  in  the  blue  color  plane. 
These  formulas  come  from  the  assumptions  that  the  green  and  blue  color  planes  are  well 
correlated  with  constant  offsets  and  that  the  rate  of  change  of  neighboring  pixels  along  a 
direction  is  also  constant  (14).  For  a  derivation  of  these  formulas,  please  see  reference  14. 
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Once  the  weights  and  directional  estimates  are  calculated,  the  final  interpolation  of  G5  can  be 
completed: 


G5  =  -  d  G  Dirs 


In  this  example,  the  missing  G  value  is  calculated  at  a  5  pixel  location.  However,  the  missing  G 
at  an  R  location  is  calculated  the  same  way,  replacing  all  5’s  in  equation  21  with  i?’s. 

Following  interpolation  of  all  missing  G  values,  BIR  values  are  interpolated  at  RIB  pixels  using 
the  full  green  color  plane.  In  this  instance,  gradient  values  are  interpolated  in  diagonal  directions, 
identified  as  NE,  SE,  SW,  and  NW,  as  shown  in  figure  9.  A  smaller,  3x3  neighborhood  is  used, 
since  a  full  green  color  plane  now  exists  for  use  in  estimating  first  order  derivatives. 


Figure  9.  3x3  Bayer  CFA  neighborhood 
with  diagonal  directions  labeled 
for  BIR  interpolation  at  RIB  pixel. 

As  an  example,  equations  for  B  interpolation  at  an  R  pixel  are  given  with  reference  to 
figure  3  (14): 

B^e  =  \B1  -  53|  +  |G5  -  G3| 

Bse  =  |51  -  B9\  +  |G5  -  G9\ 

Bsw  =  |53  -  Bl\  +  |G5  -  Gl\ 

B^=\B9-Bl\  +  \G5-Gl\ 

Dirs  =  {NE,SE,SW,NW} 

1  ,  (24) 

co^  = -  a  e  Dirs 

l  +  B, 
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^NE 

^SE 

^SW 

^NW 


=  53  + 


=  59  + 


=  57  + 


=  51  + 


G5-G3 

2 

G5-G9 

2 

G5-G7 

2 

G5-G1 

2 


55  = 


_ 

d 


d  e  Dzra 


(25) 


(26) 


Interpolation  of  5  at  a  5  pixel  is  carried  out  in  a  similar  manner.  The  remaining  R  and  5  values 
are  interpolated  in  a  similar  manner  as  the  missing  G  values,  making  full  use  of  the  previously 
interpolated  5/5  values  {14).  Proceeding  in  this  way  allows  for  a  better  estimate  of  edge 
directions  in  the  red  and  blue  color  planes  than  can  normally  be  achieved  by  using  only 
measured  values. 


Perhaps  the  best  attribute  of  this  method  of  interpolation  is  its  simplicity.  Having  the  adaptive 
nature  of  the  algorithm  encoded  within  the  arithmetic  prevents  excessive  branching,  allowing  for 
better  pipelining  and  thus  higher  speed. 


3.  Common  Demosaicing  Artifacts 


Because  sampling  a  scene  using  a  CCD  with  a  Bayer  pattern  CFA  measures  only  33%  of  the 
information  of  the  original  scene,  several  artifacts  occur  as  a  result  of  demosaicing.  Two  of  the 
most  common  are  false  coloring  and  zippering. 

3.1  False  Color  Artifact 

A  frequent  and  unfortunate  artifact  of  CFA  demosaicing  is  what  is  known  as  false  coloring.  This 
artifact  typically  manifests  itself  along  edges,  where  abrupt  or  unnatural  shifts  in  color  occur  as  a 
result  of  misinterpolating  across,  rather  than  along,  an  edge.  Figure  10  shows  three  images 
demosaiced  with  bilinear  interpolation  with  examples  of  false  colors.  Image  (a)  has  an 
alternating  pattern  of  red  and  blue  highlights  moving  along  the  left  edge  of  the  windshield,  along 
with  some  red  and  blue  highlights  on  brighter  portions  of  the  mirror.  Image  (b)  shows  another 
view  of  the  truck’s  windshield,  where  straight  lines  visible  through  the  windshield  appear  as 
alternating  red  and  yellow  pixels.  Image  (c)  shows  false  coloring  amidst  high  frequency 
information  in  the  Ford  logo’s  lettering. 
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(a) _ (b) _ 


Figure  10.  Three  images  depicting  the  false  color  demosaicing  artifact.  Image  (a)  shows  the 
comer  of  a  tmck  with  false  coloring  on  the  side  mirror  and  along  the  edge  of  the 
windshield.  Image  (b)  shows  the  window  of  a  tmck  with  false  coloring  along  the 
edges  of  the  windshield  and  along  edges  showing  through  the  windshield.  Image 
(c)  depicts  the  tmcks  insignia  with  false  coloring  amongst  the  high  frequency 
information  contained  within. 

Several  methods  exist  for  preventing  and  removing  this  false  coloring.  Smooth  hue  transition 
interpolation,  which  was  reviewed  in  section  2,  is  used  during  the  demosaicing  to  prevent  false 
colors  from  manifesting  themselves  in  the  final  image.  However,  other  algorithms  exist  that  can 
remove  false  colors  after  demosaicing.  These  have  the  benefit  of  removing  false  coloring 
artifacts  from  the  image  while  using  a  more  robust  demosaicing  algorithm  for  interpolating  the 
red  and  blue  color  planes. 


3.2.  Zippering  Artifact 


Another  side  effect  of  CFA  demosaicing,  which  also  occurs  primarily  along  edges,  is  known  as 
the  zipper  effect.  Simply  put,  zippering  is  another  name  for  edge  blurring  that  occurs  in  an  on/off 
pattern  along  an  edge.  Figure  1 1  shows  three  images  demosaicked  with  bilinear  interpolation 
featuring  the  edge  blurring  zipper  effect.  Image  (a)  features  a  truck  with  zippering  along  the 
upper  edge  of  the  grill  and  also  zippering  along  edges  within  the  headlight.  Image  (b)  features  a 
person  with  zippering  along  the  stripes  in  his  shirt  as  well  as  zippering  along  the  fence  poles  in 
the  background  of  the  image.  Image  (c)  shows  a  license  plate  with  zippering  along  its  six 
characters  and  more  zippering  along  the  upper  edge  of  the  bumper. 


Figure  11.  Three  images  depicting  the  zippering  artifact  of  CFA  demosaicing.  (a)  features  a  truck  with 
heavy  zippering  along  edges  of  the  grill  and  headlights,  (b)  features  a  person  with  zippering 
along  his  shirt’s  stripes  and  on  the  fence  poles  in  the  background,  (c)  features  a  license  plate 
with  zippering  along  its  numbers  as  well  as  zippering  along  the  edges  of  the  bumper. 
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This  effect  occurs  when  the  demosaicing  algorithm  averages  pixel  values  over  an  edge, 
especially  in  the  red  and  blue  planes,  resulting  in  its  characteristic  blur.  The  best  methods  for 
preventing  this  effect  are  the  various  algorithms  which  interpolate  along,  rather  than  across 
image  edges.  Pattern  recognition  interpolation,  adaptive  color  plane  interpolation,  and 
directionally  weighted  interpolation  all  attempt  to  prevent  zippering  by  interpolating  along  edges 
detected  in  the  image. 


4.  Post  Demosaicing  Artifact  Suppression 


In  section  2,  a  demosaicing  algorithm,  known  as  smooth  hue  transition  interpolation,  was 
introduced,  which  aims  to  reduce  the  occurrence  of  false  coloring  in  the  demosaiced  image. 
However,  several  post-processing  techniques  exist  that  enforce  similar  smooth  hue  constraints  in 
the  demosaiced  image  rather  than  the  mosaiced  raw  data.  In  this  way,  more  robust  interpolation 
techniques  can  be  used  to  interpolate  the  red  and  blue  channels  while  still  reducing  the  overall 
appearance  of  false  colors.  We  summarize  two  such  techniques  in  this  section. 

4.1  Local  Color  Ratio  Based  Post  Processing 

Similar  to  the  smooth  hue  transition  algorithm,  local  color  ratio  post  processing  is  based  on  the 
normalized  color  ratio  model  defined  in  reference  8.  The  main  goal  of  this  algorithm  is  to  correct 
unnatural  changes  in  hue  by  smoothing  the  color  ratio  planes  (75). 

For  the  moment,  let  R(x,y),  G(x,y),  and  B(x,y)  be  the  functions  representing  the  red,  green,  and 
blue  color  planes,  respectively,  where  (x,y)  is  the  pixel  at  the  x-th  column  and  the  y-th  row.  The 
first  part  of  the  algorithm  involves  adjusting  the  green  color  plane  based  on  the  newly  completed 
red  and  blue  color  planes.  It  is  important  to  note  that  this  operation  is  performed  only  on  RIB 
pixels.  Provided  as  an  example  is  the  equation  used  to  adjust  the  G  value  at  a  5  pixel  (75): 

^  =  {{p  + 1,  q),  (p,  q  + 1),  {p  - 1,  q),  {p,  q  - 1)} 

G{p,  q)  =  -/3  +  [B{p,  q)  +  /3Y  meanl 


where  p  is  a  nonnegative  constant,  as  defined  by  the  normalized  color  ratio  model  (5).  This  can 
easily  be  modified  for  use  on  an  R  pixel,  by  exchanging  B(x,y)  for  R(x,y). 


The  next  step  is  to  perform  a  similar  process  for  the  B/R  values  at  RIB  pixels.  Provided  as  an 
example  is  the  equation  for  the  adjustment  of  the  B  value  at  an  R  pixel  (75): 


4'  =  {(jP  +  l,^  +  l),(jP-hl,g-l),(;i-l,^  +  l),(jP-l,g-l)} 
B{p,  q)  =  -P  +  [g{p,  q)  +  PY  ' 


(28) 
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where  |3  has  the  same  value  as  before.  This  is,  of  course,  easily  modified  for  adjusting  the  R 
value  at  a  5  pixel.  Adjustment  of  B/R  values  at  G  pixels  is  similar,  except  that  it  uses  the  ^ 
neighborhood  from  the  adjustment  of  the  G  values. 

Testing  shows  that  this  method  performs  well  when  removing  false  colors.  Moreover,  empirical 
evidence  suggests  increasing  p  results  in  a  higher  quality  image  in  terms  of  MSB  and  PSNR  (5). 
However,  increasing  p  too  far  increases  the  risk  of  overflow  errors. 

4.2  Median  Filtering 

Unlike  methods  such  as  smooth  hue  transition  and  local  color  ratio  post  processing,  median 
filtering  works  by  acting  on  color  differences,  rather  than  color  ratios.  Given  the  image  model 
G=B+k  or  G=R+j  for  k,j  constant  in  a  local  neighborhood,  we  get  that  B-G=  -k  and  R-G=  -j,  or 
the  implication  that  the  color  differences  are  constant  within  a  local  neighborhood  of  the  pixel  in 
question.  As  such,  we  can  adjust  a  pixel’s  value  by  imposing  this  constant  color  difference  on  it. 

The  first  step  is  to  gather  all  the  color  difference  values  over  a  square  neighborhood  around  the 
pixel.  In  many  applications,  a  5x5  neighborhood  is  used  {16,  17)  though  a  smaller  3x3 
neighborhood  works  as  well  {4).  Once  the  differences  are  gathered,  their  median  is  calculated, 
and  then  used  as  an  approximation  of  what  the  current  pixel’s  color  difference  should  be. 

Assume  we  are  median  filtering  the  R/B  channel  at  the  pixel  (p,q)  and  that  ^  is  the  set  of  points 
within  the  5x5  neighborhood  surrounding  the  point  (p,q).  Then  the  equations  are  as  follows  (77): 

R{p,q)=  G{p, q)  +  median{R{i,  j) -  G{i, y)} 

^29) 

B{p,  q)  =  G{p,  q)  +  median{B{i,  j)  -  G{i,  y)} 

(>jH 

This  method  can  occasionally  add  its  own  artifacts  when  used  indiscriminately.  Thus,  it  is  best 
used  when  performed  only  on  edge  pixels  (16). 


5.  Quality  Assessment 


The  quality  of  a  demosaicing  algorithm  is  assessed  by  measuring  how  accurately  it  interpolates 
the  full  image.  Typically  this  is  performed  by  comparing  a  reference  image  to  an  image 
interpolated  from  a  Bayer  pattern  subsampling  of  the  reference  image.  This  can  be  done  in  the 
RGB  colorspace  to  which  the  image  already  belongs  or  by  transforming  the  image  into  an 
alternate  color  space,  such  as  the  perceptually  uniform  International  Commission  on 
Illumination’s  L*  a*  b*  color  space  (CIELAB). 
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5.1  Quality  Assessment  with  a  Reference 


The  simplest  teehniques  for  measuring  the  quality  of  a  demosaieing  algorithm  are  those  that  use 
a  reference  image.  In  this  case,  a  reference  image  is  scanned  from  a  fdm  image.  This  reference  is 
then  subsampled  in  such  a  way  so  as  to  replicate  raw  data  straight  from  the  CCD  with  the  Bayer 
Pattern  CFA.  This  data  is  then  demosaiced  and  compared  to  the  reference.  Standard  techniques 
for  quality  assessment  with  a  reference  include  Color  Mean  Squared  Error  (CMSE),  Color  Peak 
Signal-to-Noise  Ratio  (CPSNR),  and  the  International  Commission  on  Illumination’s  L*  a*  b* 
color  difference  (CIELAB  AE)  {14). 


5.1.1  Color  Mean  Squared  Error  and  Color  Peak  Signal-to-Noise  Ratio 

CMSE  and  CPSNR  are  very  simple  techniques.  Calculating  CMSE  involves  first  calculating  the 
squared  difference  between  the  reference  image  and  demosaiced  image  at  each  pixel  and  for  each 
color  channel.  These  are  then  summed  and  divided  by  three  times  the  area  of  the  image.  CPSNR 
is  then  calculated  using  CMSE.  The  relevant  equations  are  shown  below  {18)'. 


W  H 


CMSE  = 


CPSNR  =  mog 


3WH 
^  255"  ^ 

yCMSEj 


(30) 


where  /  is  the  reference  image,  I  is  the  demosaiced  image,  and  W  and  H  represent  the  width  and 
height  of  the  image,  respectively. 

Both  of  these  methods  fail  to  distinguish  the  case  where  the  demosaiced  image  is  only  slightly 
different  from  the  reference  over  many  pixels  from  the  case  where  the  demosaiced  image  is 
vastly  different  from  the  reference  image  over  few  pixels.  However,  the  second  case  represents 
an  image  with  very  severe  artifacts,  and  thus  a  poorer  quality  image  {18). 


5.1.2  CIELAB  AE 


Since  CMSE  and  CPSNR  given  as  in  section  5.1.1  do  not  equate  to  the  human  perception  of 
color  difference,  a  perceptually  uniform  color  space  must  be  used.  In  this  case,  the  Euclidean 
distance  between  colors  represented  in  the  CIELAB  perceptually  uniform  colorspace  are 
calculated  as  AE.  Calculating  AE  requires  first  transforming  it  from  the  RGB  colorspace  into  the 
CIELAB  colorspace,  and  then  calculating  the  average  Euclidean  distance  between  the  pixel 
colors  in  this  space.  The  equation  is  given  by  {14) 


AE  = 


1 

WH 


W  H 


/=1  y=l 


(31) 


where  again  W  and  H  represent  width  and  height  respectively,  and  I  and  /  represent  the 
reference  and  demosaiced  image,  respectively,  both  now  in  the  CIELAB  colorspace. 
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Though  this  method  better  estimates  the  eolor  differences  as  perceived  by  the  human  visual 
system,  it  still  does  not  give  a  good  global  estimate  of  image  quality.  Like  CPSNR,  this  method 
cannot  distinguish  between  small  differences  over  the  whole  image  and  severe  differences  over 
part  of  the  image. 

5.1.3  Issues  Surrounding  CMSE,  CPSNR,  and  AE 

The  main  issue  with  these  techniques  is  that  they  do  not  give  a  good  global  approximation  of 
image  quality.  Demosaiced  images  with  similar  or  equal  values  in  any  of  these  three  measures 
are  not  necessarily  similar  or  the  same,  since  these  values  are  calculated  as  averages  over  the 
entire  image. 

Another  major  detriment  to  these  techniques  is  that  they  each  require  a  reference  image.  Since 
the  quality  of  an  anonymous  image  is  rarely  known,  finding  reference  images  can  be  very 
difficult.  Methods  that  compare  demosaiced  images  without  a  reference  would  then  be  more 
applicable  in  real-world  situations  where  references  frequently  do  not  exist. 

5.2  No-reference  Quality  Assessment 

As  previously  discussed,  there  are  inherent  disadvantages  to  using  quality  assessment  techniques 
that  require  a  reference  image.  In  response,  two  new  no-reference  techniques  are  proposed  in  this 
report.  Before  the  new  techniques  are  proposed,  a  current  technique  known  as  Blur  Measure  {19) 
will  be  introduced,  which  measures  the  amount  of  blur  along  an  image  edge.  Then,  the  first  new 
technique,  which  is  an  extension  of  the  Blur  Measure  technique,  will  be  proposed.  A  second  new 
technique  will  then  be  proposed,  which  operates  by  measuring  the  false  coloring  along  an  edge 
within  the  image.  Using  these  two  new  techniques,  the  quality  of  a  demosaicing  algorithm’s 
results  will  be  measured  based  on  that  algorithm’s  edge  reconstruction  abilities. 

5.2.1  Blur  Measure 

A  simple  measure  of  blurring  in  an  image  proposed  in  reference  19  involves  calculating  what  is 
known  as  an  edge’s  width.  Averaging  the  edge  width  over  the  entire  image  produces  what  is 
defined  as  the  blur  measure. 

In  order  to  calculate  blur  measure,  all  edge  pixels  in  an  image  must  be  identified,  using  either  the 
Sobel  or  Prewitt  filters  in  conjunction  with  a  threshold  to  reduce  noise.  Once  the  edge  pixels  are 
found,  the  edge  width  is  calculated  by  first  getting  the  profile  of  the  image  in  the  vertical 
direction  passing  through  the  edge  pixel.  The  width  of  the  edge  is  then  found  by  calculating  the 
distance  between  the  locations  of  the  two  local  extrema  closest  to  the  edge  pixel.  This  process  is 
repeated  for  all  edge  pixels,  with  the  blur  measure  for  the  image  being  defined  as  the  average 
edge  width  over  the  entire  image. 
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An  example  edge  profile  is  shown  in  figure  12.  The  edge  pixel  and  corresponding  local  extrema 
are  marked  on  the  graph  with  circles.  The  center  circle  marks  the  edge  pixel.  The  right  and  left 
circles  mark  the  extrema.  The  left  extremum  is  located  at  x  =  4  and  the  right  extremum  at  x  =  6. 
Thus,  edge  width  =  6-^  =  2  in  this  example.  If  this  is  the  only  edge  pixel  in  the  image,  then  blur 
measure  =  2/1  =2. 


Figure  12.  An  example  edge  profile  with  the  edge  pixel  and  loeal 
extrema  marked  with  eircles.  The  eenter  circle  marks 
the  edge  pixel  and  the  outer  two  circles  mark  the  extrema. 

5.2.2  Edge  Slope  Measure 

For  the  first  new  no-reference  technique,  the  concept  of  edge  width  is  expanded  to  define  the 
notion  of  edge  slope.  Moreover,  rather  than  using  only  the  edge  profile  in  either  the  horizontal  or 
vertical  direction,  the  edge  profile  is  taken  along  the  edge’s  gradient. 

The  general  procedure  for  finding  edge  slope  is  to  first  find  the  edge  points  in  an  image,  using 
again  either  the  Sobel  or  Prewitt  filters,  as  well  as  a  threshold  large  enough  to  filter  out  noise  and 
insignificant  edges  but  small  enough  to  preserve  the  actual  edges  in  the  image.  Edges  are  found 
in  this  report  using  the  process  outlined  in  reference  20.  For  a  uniform  test,  only  the  edge  pixels 
common  to  all  images  are  tested  so  as  to  ensure  only  true  edges  are  tested.  Using  the  gradient 
information  gathered  from  the  edge  detection  stage,  an  edge  profile  is  taken  at  an  edge  point  in 
the  direction  of  that  edge  point’s  gradient.  The  local  extrema  are  then  found  on  this  edge  profile 
in  the  same  manner  as  for  blur  measure,  and  edge  width  is  again  defined  as  the  distance  between 
the  extrema.  Edge  height  is  now  defined  to  be  the  difference  between  the  values  of  these 
extrema,  and  edge  slope  defined  to  be  the  ratio  between  the  edge  height  and  edge  width.  Edge 
slope  measure  is  then  the  average  absolute  value  of  edge  slope  over  all  edge  points  in  the  image. 
For  the  example  edge  shown  in  figure  12,  the  edge  width  would  be  Ax  and  the  edge  height  would 
be  Ay.  The  edge  slope  would  then  be  Ay/  Ax.  The  necessary  equations  are 
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(32) 


^  =  {CommonEdgePixels} 

■  .\_  EdgeHeight{i,j) 

’  Edge  Width{i,  j  ) 

Y,EdgeSlope{i,j) 

('  ■)  Z' 

EdgeSlopeMeasure  =  - 

cardy^) 

where  edge  height  and  width  are  defined  as  previously  discussed  and  card{(^)  is  the  cardinality 
of  or  equivalently  the  number  of  points  in 

The  main  intention  of  this  assessment  technique  is  to  give  a  relative  measure  of  sharpness  of 
image,  for  use  in  comparing  the  edge  reconstruction  of  various  demosaicing  algorithms.  Thus 
when  edge  slope  measure  is  calculated  for  a  set  of  images  demosaiced  from  the  same  raw  data, 
the  image  with  the  highest  edge  slope  measure  will  have  the  highest  sharpness  and  therefore 
represent  the  demosaicing  algorithm  with  the  best  edge  reconstruction. 

An  example  edge  profile  is  shown  in  figure  12.  The  edge  pixel  and  corresponding  local  extrema 
are  marked  on  the  graph  with  circles.  The  center  circle  marks  the  edge  pixel.  The  right  and  left 
circles  mark  the  extrema.  The  left  extremum  is  located  at  (x,y)  =  (4,245)  and  the  right  extremum 
at  (x,y)  =  (6,35).  Thus,  edge  slope  =  (35  -  245)/  (6-4)  =  -105.  If  this  is  the  only  edge  pixel  in  the 
image,  then  the  edge  slope  measure  =  |-105|/1  =  105. 


5.2.3  False  Color  Measure 


The  next  important  demosaicing  artifact  we  seek  to  identify  is  the  false  color  artifact  defined  in 
section  3.  As  mentioned  previously,  false  colors  occur  where  there  are  abrupt  changes  in  color, 
usually  along  an  edge  in  the  image.  Using  the  constant  color  difference  model  G  =  R+k  =  B+j  for 
k,j  constant  in  a  local  neighborhood,  the  false  coloring  of  an  image  is  measured  by  divergence 
from  this  model  by  way  of  MSB.  Given  that  false  coloring  occurs  typically  along  edges,  this 
method  is  only  performed  along  edges. 


The  algorithm  involves  first  finding  the  edge  pixels  in  the  set  of  images  under  consideration.  As 
with  edge  slope  measure,  only  edge  pixels  common  to  all  images  are  tested  to  enforce  a  uniform 
test.  At  each  edge  pixel,  the  median  color  difference  is  calculated  in  a  5x5  neighborhood 
centered  at  the  edge  pixel  under  consideration.  The  false  color  measure  for  the  red  channel  is 
calculated  as  follows: 


=  {CommonEdgePixels] 


dJH _ 


RFalseColorMeasure  = 


(33) 


where  Mq  is  the  median  color  difference  for  the  red  channel  at  the  point  (i,j)  and  card(Q  is  the 
cardinality  of  or  equivalently  the  number  of  common  edge  pixels.  The  false  color  measure  for 
the  blue  channel  is  calculated  analogously. 

Like  the  edge  slope  measure,  this  assessment  gives  a  relative  comparison  of  demosaiced  image 
quality.  Given  that  is  technique  measures  divergence  from  the  expected  pattern,  the  better 
demosaicing  algorithm  will  produce  an  image  with  a  lower  false  color  measure. 

5.3  Performance  Analysis  of  the  Demosaicing  Algorithms  with  New  Techniques 

Using  these  two  tests  in  conjunction  with  one  another  should  give  a  full  comparison  of  any  set  of 
demosaicing  algorithms  performed  on  the  same  set  of  raw  data.  The  edge  slope  measure  is  used 
to  test  the  overall  sharpness  of  each  of  the  three  color  channels,  thus  estimating  the  relative  edge 
reconstruction  accuracy  of  each  demosaicing  algorithm.  The  false  measure  is  performed  on  the 
G-R  and  G-B  color  difference  planes,  estimating  deviations  from  the  established  constant  color 
difference  image  model,  and  therefore  estimating  the  R  and  B  channel  reconstruction  of  each 
demosaicing  algorithm.  Thus,  the  overall  quality  of  a  demosaicing  algorithm  is  its  relative  ability 
to  minimize  the  common  blurring  and  false  color  artifacts. 

For  testing  purposes,  we  used  real  CFA  data  collected  at  the  Ft.  Belvoir  Night  Vision  Lab  on 
June  24,  2009.  Twenty  images  from  each  of  eight  image  sets  were  demosaiced  using  the 
following  algorithms: 

•  linear  interpolation  (Linear), 

•  cubic  interpolation  (Cubic), 

•  linear  interpolation  with  smooth  hue  transition  (Linear  w/SH), 

•  pattern  recognition  interpolation  with  smooth  hue  transition  (PR  w/SH), 

•  adaptive  color  plane  interpolation  (ACP), 

•  directionally  weighted  interpolation  (DW), 

•  directionally  weighted  interpolation  with  local  color  ratio  based  post  processing  with 
p=128  (DW  w/LCR),  and 

•  directionally  weighted  interpolation  with  median  filtering  (DW  w/MF). 

Four  of  the  image  sets  featured  a  white  pickup  truck  at  different  distances.  The  other  four  image 
sets  featured  two  men  standing  at  progressively  further  distances.  Since  no  reference  data  existed 
for  these  images,  they  were  analyzed  with  the  new  proposed  algorithms.  They  were  analyzed 
with  the  following: 

•  the  edge  slope  measure  test  in  the  G  channel, 

•  the  edge  slope  measure  test  in  the  R  channel. 
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•  the  edge  slope  measure  test  in  the  B  ehannel, 

•  the  false  color  measure  test  in  the  G-R  color  plane,  and 

•  the  false  color  measure  test  in  the  G-B  color  plane. 

The  results  are  presented  as  line  graphs  showing  the  average  measurement  for  each  demosaicing 
algorithm  in  each  range.  For  subjective  analysis,  a  large  number  of  image  samples  are  also  given. 


6.  Results  and  Discussion 


6.1  Edge  Slope  Measure  Results 

The  results  for  the  edge  slope  measure  test  are  shown  in  figures  13-15.  The  results  for  the  green 
channel  are  shown  in  figure  13,  the  results  for  the  red  channel  are  shown  in  figure  14,  and  the 
results  for  the  blue  channel  are  shown  in  figure  15.  Each  figure  is  split  into  results  for  the  truck 
image  sets  and  results  for  the  people  image  sets. 

In  figure  13,  it  is  clear  that  AGP  interpolation  produced  the  sharpest  green  color  plane  from  the 
raw  CFA  data.  It  had  the  highest  average  edge  slope  measure  in  both  the  truck  and  people  image 
sets  across  all  ranges  of  each.  The  DW  and  DW  w/MF  interpolations  ranked  second  highest  (they 
produced  the  same  average  slope,  since  the  median  filter  defined  previously  does  not  change  the 
green  values)  in  both  image  sets.  Surprisingly,  cubic  interpolation  achieved  the  next  highest 
average  edge  slope  measure,  despite  being  a  simple  kernel-based  interpolation  scheme.  The  local 
color  ratio  based  post  processor  (in  DW  w/LCR),  which  unlike  median  filtering  (in  DW  w/MF) 
changes  values  in  the  green  channel,  brought  an  overall  reduction  in  the  edge  slope  of  the  DW 
interpolation’s  green  channel.  The  linear  interpolation  performed  the  worst,  as  expected,  since  it 
operates  by  averaging  the  values  of  nearby  pixel,  which  creates  a  blurred  image  as  a  result. 
Smooth  hue  transition  does  not  affect  the  green  channel  since  by  definition  it  does  not  operate  on 
these  values,  so  the  Linear  and  Linear  w/SH  interpolation  algorithms  produced  the  same  results 
in  the  green  channel. 
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Figure  13.  Average  edge  slope  measure  for  the  green  channel:  (a)  The  performance  of  the  various  algorithms  for 
the  images  sets  showing  a  truck  at  four  increasing  distances  and  (b)  the  performance  of  the  algorithms 
for  the  image  sets  showing  two  people  standing  at  increasing  distances. 


In  figure  14,  it  may  be  surprising  to  see  that  the  Linear  w/SH  interpolation  performed  best  on  the 
red  channel.  In  view  of  the  fact  that  plain  linear  interpolation  again  performed  the  worst  on  the 
red  channel,  this  shows  the  extent  to  which  exploiting  interchannel  correlations  can  improve  the 
quality  of  the  demosaiced  image.  This  is  further  shown  with  the  improvements  attained  by  post 
processing  the  DW  interpolation.  Both  the  median  filter  (in  DW  w/MF)  and  local  color  ratio 
based  post  processor  (in  DW  w/LCR)  improved  the  edge  slope  in  the  red  channel.  Both  the  ACP 
and  cubic  interpolations  performed  worse  than  expected,  given  the  results  in  the  green  channel. 
PR  interpolation  performed  well  in  this  channel,  which  is  likely  the  result  of  the  smooth  hue 
transition  used  to  interpolate  the  red  and  blue  channels. 
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Figure  14.  Average  edge  slope  measure  for  the  red  channel:  (a)  The  performance  of  the  various  algorithms  for  the 
images  sets  showing  a  truck  at  four  increasing  distances  and  (b)  the  performance  of  the  algorithms  for 
the  image  sets  showing  two  people  standing  at  increasing  distances. 

In  figure  15,  it  is  immediately  obvious  the  effect  median  filtering  had  on  the  edge  slope  in  the 
blue  channel,  with  the  DW  w/MF  interpolated  images  being  at  least  five  units  higher  than  any 
other  interpolation  technique.  Other  than  this,  there  were  few  differences  between  results  for  the 
blue  channel  and  results  for  the  red  channel. 


(a)  (b) 

Figure  15.  Average  edge  slope  measure  for  the  blue  channel:  (a)  The  performance  of  the  various  algorithms  for 


the  images  sets  showing  a  truck  at  four  increasing  distances  and  (b)  the  performance  of  the  algorithms 
for  the  image  sets  showing  two  people  standing  at  increasing  distances. 
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One  thing  to  note  in  these  tests  is  that  the  performance  of  these  algorithms  decreased  with  the 
range  in  the  truck  image  sets.  No  generalization  existed  for  performance  changes  with  range  for 
the  people  image  sets,  as  the  trends  were  different  in  each  color  channel.  This  may  be  due  to  the 
fact  that  manmade  objects  exhibit  sharp  edges  while  people  and  natural  objects  exhibit  smoother 
edges.  One  thing  worth  noting,  however,  is  that  in  the  blue  and  red  color  channels,  techniques 
that  used  the  smoothly  changing  color  ratio  and  color  difference  models  performed  better.  The 
two  post-processing  techniques  both  showed  improvement  in  the  red  and  blue  color  channels  in 
terms  of  edge  slope,  despite  a  decrease  in  performance  in  the  green  channel  using  the  local  color 
ratio  based  post  processing  (in  DW  w/LCR). 

6.2  False  Color  Measure  Results 

The  results  for  the  false  color  measure  test  are  shown  in  figures  16-17.  The  results  for  the  truck 
image  sets  are  shown  in  figure  16  and  the  result  for  the  people  image  sets  in  figure  17.  Each 
figure  splits  the  results  into  results  for  the  red  channel  and  results  for  the  blue  channel. 

In  figure  16,  we  see  that  the  DW  interpolation  with  the  two  post-processing  techniques 
performed  the  best  on  both  the  red  and  blue  channels,  followed  by  Linear  w/SH  and  then  the 
regular  DW  interpolation.  Cubic  interpolation  performed  the  poorest,  followed  by  linear 
interpolation  and  then  adaptive  color  plane  interpolation.  The  false  color  measure  for  the  truck 
image  sets  decreased  with  range  initially,  and  then  increased  slightly  at  the  end  in  the  red 
channel,  and  continued  decreasing  in  the  blue  channel,  creating  a  convex  graph. 


color  measures  for  the  red  channel  and  (b)  the  average  false  color  measures  for  the  blue  channel. 
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In  figure  17  the  results  are  nearly  the  same.  The  only  difference  we  notice  is  that  initially  false 
color  measure  increased  with  the  increase  in  range,  and  then  decreased  at  the  end,  creating  a 
concave  graph.  Otherwise,  DW  interpolation  with  post  processing  performed  the  best,  and  the 
two  kernel-based  techniques  without  smooth  hue  transition  performed  the  worst.  All  other 
techniques  had  the  same  relative  performance  as  with  the  truck  image  set. 


Figure  17.  Average  false  color  measure  for  images  featuring  the  two  people  at  different  ranges:  (a)  The  average 
false  color  measures  for  the  red  channel  and  (b)  the  average  false  color  measures  for  the  blue 
channel. 


As  with  the  edge  slope  measure,  we  see  immediately  that  techniques  exploiting  interchannel 
correlations  performed  the  best,  producing  the  least  amount  of  false  colors.  Also  worth  noting  is 
that  all  interpolation  techniques  performed  worse  in  the  blue  channel  compared  to  the  red 
channel  across  all  image  sets. 

6.3  Image  Examples 

Figures  18-24  each  give  eight  image  samples,  one  for  each  demosaicing  algorithm  tested. 

Figures  18-21  give  image  samples  from  range  one  of  the  truck  image  sets  and  figures  22-24  give 
image  samples  from  range  one  of  the  people  image  sets. 

In  view  of  the  edge  slope  measure  and  false  color  measure  results  shown  in  figures  13-17  and 
the  discussion  in  sections  4.3.1  and  4.3.2,  the  subjective  quality  of  these  images  conformed  to  the 
objective  results.  We  see  in  all  cases  that  DW  interpolation  with  some  form  of  post  processing 
produced  the  least  amount  of  false  colors  as  well  as  had  the  sharpest  edges  (figures  1 8-24  (g) 
and  (h)).  The  linear  and  cubic  interpolations  produced  significant  and  highly  visible  false 
coloring  along  edges,  which  tended  to  be  blurry  in  comparison  to  other  techniques  (figures  1 8- 
24  (a)  and  (b)).  In  comparing  (a)  to  (c)  in  figures  18-24,  it  is  apparent  that  the  false  coloring  of 
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the  linear  interpolation  was  removed  when  the  smooth  hue  transition  interpolation  was  applied, 
though  the  edge  sharpness  did  not  inerease  overall.  When  eomparing  (f)  to  (g)  and  (h)  of  figures 
18-24  we  again  see  the  effeet  post  proeessing  has  on  the  presence  of  false  colors.  The  dark  edges 
of  (f)  in  figures  18-19  have  noticeable  false  color  artifacts,  which  were  mostly,  if  not  completely 
removed  by  the  post  processing. 

Of  particular  interest  are  figures  20-21,  which  feature  the  truck’s  license  plate  and  brand 
insignia.  Images  (a)-(c)  all  feature  edge  zippering  and  false  colors,  which  make  it  difficult  to 
read  the  license  plate  and  insignia.  Moving  through  to  image  (f),  both  figures  18  and  19  slowly 
become  more  readable  with  fewer  artifacts.  Finally,  images  (g)  and  (h)  are  most  easily  read  with 
the  highest  sharpness  and  the  lowest  occurrence  of  false  colors.  In  particular,  notice  the  effect 
post  processing  had  on  the  truck’s  insignia,  viewable  in  figure  21  (f)-(h).  For  reference,  the 
license  plate  number  is  W  42916. 


(d) 


(h) 


Figure  18.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  truck  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 
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Figure  19.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  truck  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 


(a) 


(b) 


(c) 


(d) 


Figure  20.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  truck  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 
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Figure  21.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  truck  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 


Figure  22.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  truck  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 
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Figure  23.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  people  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 


Figure  24.  Image  samples  demosaiced  from  the  first  image  of  the  closest  range  of  the  people  image  sets  using 
the  following  algorithms:  (a)  linear,  (b)  cubic,  (c)  linear  with  smooth  hue  transition,  (d)  pattern 
recognition  with  smooth  hue  transition,  (e)  adaptive  color  plane,  (f)  directionally  weighted, 

(g)  directionally  weighted  with  local  color  ratio  post  processing,  and  (h)  directionally  weighted 
with  median  filter  post  processing. 
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7.  Conclusions 


In  this  report,  we  proposed  two  new  techniques  for  assessing  the  relative  quality  of  images 
demosaiced  from  the  same  CFA  data  using  different  demosaicing  strategies.  The  first  measures 
the  relative  sharpness  of  the  images  by  calculating  the  average  edge  slope  along  edge  points 
common  to  all  images.  The  second  measures  the  relative  quantity  of  false  coloring  by 
approximating  the  average  squared  deviation  from  the  locally  constant  color  difference  image 
model  along  edge  points  common  to  all  images.  Following  the  presentation  of  objective  results 
using  each  technique,  a  large  number  of  image  samples  were  given  to  show  that  the  assessment 
results  were  in  accordance  with  the  quality  of  the  example  images.  Moreover,  given  the  image 
samples  and  no-reference  test  results,  we  concluded  that  DW  w/MF  provided  the  best  overall 
demosaiced  image  with  the  fewest  color  artifacts  and  sharpest  edges. 

This  report  is  intended  to  establish  a  path  forward  for  future  research  investigating  possible 
relationships  between  CFA  demosaicing  and  color  image  super-resolution.  Reference  18  shows 
that  the  quality  of  a  demosaiced  image  decreases  with  the  resolution  of  the  raw  data,  suggesting 
possible  gains  from  super-resolving  a  raw  image  sequence  before  demosaicing.  With  new 
machinery  for  comparing  image  quality  without  a  reference  image,  a  study  investigating  ties 
between  CFA  demosaicing  and  color  image  super-resolution  is  now  within  reach. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


CCD 

CFA 

CIELAB 

CIELAB  AE 

CMOS 

CMSE 

CPSNR 

MSE 

PSNR 


charged-coupled  device 
color  fdter  array 

International  Commission  on  Illumination’s  L*  a*  b*  color  space 

International  Commission  on  Illumination’s  L*  a*  b*  color  difference 

complementary  metal-oxide  semiconductor 

color  mean  squared  error 

color  peak  signal-to-noise  ratio 

mean  squared  error 

peak  signal-to-noise  ratio 
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